Displaying Objects Based On A Plurality Of Models

ABSTRACT

A system and method is provided for displaying surfaces of an object from a vantage point different from the vantage point from which imagery of the object was captured. In some aspects, imagery may be generated for display by combining visual characteristics from multiple source images and applying greater weight to the visual characteristics of some of the source images relative to the other source images. The weight may be based on the orientation of the surface relative to the location from which the image was captured and the location from which the object will be displayed.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/575,941, filed Sep. 19, 2019, which is a continuation ofU.S. patent application Ser. No. 15/692,548, filed Aug. 31, 2017, whichis a continuation of Ser. No. 14/877,368, filed Oct. 7, 2015, issued asU.S. Pat. No. 9,773,022, the disclosures of which are herebyincorporated herein by reference.

BACKGROUND OF THE INVENTION

Certain panoramic images of objects are associated with informationrelating to the geographic location and orientation from which the imagewas captured. For example, each pixel of the image may be associatedwith data that identifies the angle from the geographic location fromwhich the image was captured to the portion of the surface of an object(if any) whose appearance is represented by the visual characteristicsof the pixel. Each pixel may also be associated with depth data thatidentifies the distance from the capture location to the portion of thesurface represented by the pixel.

Three-dimensional models of the locations of surfaces appearing in theimages may be generated based on the depth data. The models may includepolygons whose vertices correspond with the surface locations. Thepolygons may be textured by projecting visual characteristics of thepanoramic image onto the model using ray tracing. A user may select avantage point from which the models may be displayed to the user.

BRIEF SUMMARY OF THE INVENTION

Aspects of the disclosure provide a system that includes one or moreprocessors, memory storing a model of the orientation and visualcharacteristics of the surface of an object relative to vantage points,and instructions executable by the one or more processors. The visualcharacteristics may include a first set of visual characteristicsrepresenting the appearance of the surface from a first vantage pointand a second set of visual characteristics representing the appearanceof the surface from a second vantage point. The instructions mayinclude: receiving a request for an image of the object from a requestedvantage point different from the first vantage point and the secondvantage point; identifying a first visual characteristic from the firstset of visual characteristics and a second visual characteristic fromthe second set of visual characteristics; determining a first weightvalue for the first visual characteristic based on the orientation ofthe surface relative to the requested vantage point and the firstvantage point; determining a second weight value for the second visualcharacteristic based on the orientation of the surface relative to therequested vantage point and the second vantage point; determining avisual characteristic of the requested image based on the first andsecond visual characteristics and the first and second weight values;and providing the requested image.

Aspects of the disclosure also provide a method of providing an imagefor display. The method may include: receiving a request for an image ofan object from a requested vantage point; accessing a model of theorientation and visual characteristics of the surface of the objectrelative to vantage points, where the visual characteristics include afirst set of visual characteristics representing the appearance of thesurface from a first vantage point and a second set of visualcharacteristics representing the appearance of the surface from a secondvantage point, the first and second vantage points being different fromthe requested vantage point; identifying a first visual characteristicfrom the first set of visual characteristics and a second visualcharacteristic from the second set of visual characteristics;determining a first weight value for the first visual characteristicbased on the orientation of the surface relative to the requestedvantage point and the first vantage point; determining a second weightvalue for the second visual characteristic based on the orientation ofthe surface relative to the requested vantage point and the secondvantage point; determining a visual characteristic of the requestedimage based on the first and second visual characteristics and the firstand second weight values; and providing the requested image for display.

Aspects of the disclosure further provide a non-transitorycomputing-device readable storage medium on which computing-devicereadable instructions of a program are stored. The instructions, whenexecuted by one or more computing devices, may cause the one or morecomputing devices to perform a method that includes: receiving a requestfor an image of an object from a requested vantage point; accessing amodel of the orientation and visual characteristics of the surface ofthe object relative to vantage points, where the visual characteristicscomprise a first set of visual characteristics representing theappearance of the surface from a first vantage point and a second set ofvisual characteristics representing the appearance of the surface from asecond vantage point, and where the first and second vantage points aredifferent from the requested vantage point; identifying a first visualcharacteristic from the first set of visual characteristics and a secondvisual characteristic from the second set of visual characteristics;determining a first weight value for the first visual characteristicbased on the orientation of the surface relative to the requestedvantage point and the first vantage point; determining a second weightvalue for the second visual characteristic based on the orientation ofthe surface relative to the requested vantage point and the secondvantage point; determining a visual characteristic of the requestedimage based on the first and second visual characteristics and the firstand second weight values; and providing the requested image for display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of a system in accordance with aspects ofthe disclosure.

FIG. 2 is a diagram of an object relative to vantage points from whichthe object may be captured and displayed.

FIG. 3 is a diagram of an ellipse that is generated based a vantagepoint and the orientation of a surface.

FIG. 4 is a diagram of a texel ellipse and a pixel ellipse.

FIG. 5 is a diagram of an occluded surface relative to vantage pointsfrom which the object may be captured and displayed.

FIG. 6 is a diagram of an extraneous surface relative to vantage pointsfrom which the object may be captured and displayed.

FIG. 7 is an example of an image that may be displayed to a user.

FIG. 8 is an example flow diagram in accordance with aspects of thedisclosure.

DETAILED DESCRIPTION Overview

The technology relates to displaying an object from a vantage point thatis different from the vantage point from which imagery of the object wascaptured. For instance, two or more panoramic images may capture anobject from two different vantage points, and a user may request animage of the object from a location between the two capture points. Thesystem may generate the user-requested image by blending correspondingfragments of the images together in proportion to the likelihood of thefragment being a visually-accurate representation of the correspondingsurface of the object. By way of example, when generating theuser-requested image, the system may calculate a quality value that isbased on the relationship of the capture location and user-requestedpoint to the orientation of the surface of the object to be displayed.When the fragments are blended, more weight may be applied to fragmentshaving better quality values than the other fragments.

By way of illustration, FIG. 2 shows two different vantage points fromwhich two source images of a car were captured. In this example, theangle from which the front of the car is captured is relativelyorthogonal in the first source image and relatively acute in the secondsource image. Conversely, the angle from which the side of the car iscaptured is relatively acute in the first image and relativelyorthogonal in the second image. The figure also shows a vantage pointselected by a user to view the car.

In order to display the object from the user-selected vantage point, thesystem may generate a three-dimensional (3D) model of all of thesurfaces captured in each source image. For instance, a laser rangefinder may have been used to prepare a depth map, which was in turn usedto prepare a source model that includes a mesh of polygons whosevertices correspond with the locations of points along the surfaces ofthe object.

The source model associated with each source image may also identify thelocation of the capture point relative to the model, and the system mayuse that location to project the visual information captured in thesource image onto the model. The 3D model associated with one sourceimage may be substantially identical to the 3D model associated withanother source image with respect to surface locations, but the visualcharacteristics of textures projected onto the models may be differentdepending on the angle from which the surfaces was captured.

When determining the visual characteristics of the pixels of the imageto be displayed to the user, the system may use ray tracing and thelocation of the user-requested vantage point to identify the location atwhich a ray extending through each displayed pixel intersects thetextures of the model, e.g., a texel. The system may blend the texelsfrom the different source images together to determine the visualcharacteristics of the displayed pixel (e.g., hue, saturation andbrightness).

When the texels from the source images' models are blended together,greater weight may be applied to the texel from one source image over atexel from another source image. The weight may be based on a qualityvalue that reflects the likelihood of the texel being an accuraterepresentation of the visual characteristics of object to be displayed.

In at least one aspect, the quality value may depend on the resolutionof the displayed pixels relative to the resolution of the texels. Forinstance, optimum quality may be defined to occur when there is a singletexel for each displayed pixel. Conversely, low quality may be definedto occur when there are many texels associated with a single pixel(which may occur when a texture of a surface was captured straight onbut is viewed at a grazing angle) or there are many pixels associatedwith a single texel (which may occur if the texture of a surface wascaptured at a grazing angle but is viewed straight on).

The quality value of a texel may be calculated based on the location ofthe user-defined and capture vantage points relative to the orientationof the surface to be displayed. By way of illustration, FIG. 3 shows thelocation of two points: a vantage point and a point on the surface ofthe model to be displayed to the user. The figure also shows an ellipserepresenting the intersection of a cone and a plane. The plane reflectsthe orientation of the surface relative to the vantage point, e.g., aplane defined by the vertices of the source model polygon containing thetexel. The cone is centered on a line extending from the vantage pointto the surface point (the “vantage/surface line”). The extent to whichthe ellipse is stretched is related to the orientation of the surfaceand the angle from which the surface is viewed from the vantage point.If the vantage/surface line is perfectly orthogonal to the orientationof the surface, the ellipse will be a circle. As the solid angle of thevantage/surface line becomes more acute relative to the orientation ofthe surface, the ellipse will become more stretched and the ratio of theellipse's major axis relative to the minor axis will increase.

The quality value of a texel may be determined based on the differencesbetween an ellipse associated with the capture point (the “texelellipse”) and an ellipse associated with the user-requested vantagepoint (the “pixel ellipse”). FIG. 4 provides an example of a texelellipse and a pixel ellipse. The quality value may be calculated fromthe ratio of the radius of the pixel ellipse relative to the radius ofthe texel ellipse at an angle that yields the greatest difference inlength between the two radii.

Once the quality value has been calculated for each texel, the qualityvalue may be applied as weights during blending. For instance, if threesource images were used to identify three texels T₁, T₂ and T₃, theoutput may be calculated as (w₁T₁+w₂T₂ +w₃T₃)/(w₁+w₂+w₃), where w_(n) isequal to the quality value of the texel. The weights may also be appliedin other ways.

The system may also use weight values to address occlusion. As shown inFIG. 5, a ray used to determine the characteristics of a displayed pixelmay extend through surfaces of the object that were captured in thesource images but would be obstructed from view at the user-requestedvantage point. The system may render both front and back facing surfacessuch that the weight for each back facing surface is set to zero. Thesystem may also generate the image so that the surface that is closestto the viewer within each source image is selected for display, thusallowing the system to blend textures from both models together withoutcalculating depth.

Artifacts may also be addressed with weight values. For example and asshown in FIG. 6, discontinuities in depth data may cause gaps insurfaces to be incorrectly modeled as the surface of an object. Ratherthan removing such non-existent surfaces from a model, the system maydetermine a quality value for a texel of the non-existent surface. Ifthe angle of the ray is relatively orthogonal to the orientation of thenon-existing surface, the quality value of the texel may be very lowcompared to the quality value of a texel on another surface capturedfrom a different vantage point. However, if the angle from which thenon-existing surface is viewed is relatively parallel to the orientationof the non-existing surface, the quality value of the texel on thatsurface may be relatively high.

The system may be used to display the object to the user from thevantage point requested by the user. In that regard and as shown in FIG.7, a user may be able to view the object from vantage points other thanthe vantage points from which the source imagery was captured.

Example Systems

FIG. 1 illustrates one possible system 100 in which the aspectsdisclosed herein may be implemented. In this example, system 100 mayinclude computing devices 110 and 120. Computing device 110 may containone or more processors 112, memory 114 and other components typicallypresent in general purpose computing devices. Although FIG. 1functionally represents each of the processor 112 and memory 114 as asingle block within device 110, which is also represented as a singleblock, the system may include and the methods described herein mayinvolve multiple processors, memories and devices that may or may not bestored within the same physical housing. For instance, various methodsdescribed below as involving a single component (e.g., processor 112)may involve a plurality of components (e.g., multiple processors in aload-balanced server farm). Similarly, various methods described belowas involving different components (e.g., device 110 and device 120) mayinvolve a single component (e.g., rather than device 120 performing adetermination described below, device 120 may send the relevant data todevice 110 for processing and receive the results of the determinationfor further processing or display).

Memory 114 of computing device 110 may store information accessible byprocessor 112, including instructions 116 that may be executed by theprocessor. Memory 114 may also include data 118 that may be retrieved,manipulated or stored by processor 112. Memory 114 may be any type ofstorage capable of storing information accessible by the relevantprocessor, such as media capable of storing non-transitory data. By wayof example, memory 114 may be a hard-disk drive, a solid state drive, amemory card, RAM, DVD, write-capable memory or read-only memory. Inaddition, the memory may include a distributed storage system wheredata, such as data 150, is stored on a plurality of different storagedevices which may be physically located at the same or differentgeographic locations.

The instructions 116 may be any set of instructions to be executed byprocessor 112 or other computing device. In that regard, the terms“instructions,” “application,” “steps” and “programs” may be usedinterchangeably herein. The instructions may be stored in object codeformat for immediate processing by a processor, or in another computingdevice language including scripts or collections of independent sourcecode modules, that are interpreted on demand or compiled in advance.Functions, methods and routines of the instructions are explained inmore detail below. Processor 112 may be any conventional processor, suchas a commercially available CPU. Alternatively, the processor may be adedicated component such as an ASIC or other hardware-based processor.

Data 118 may be retrieved, stored or modified by computing device 110 inaccordance with the instructions 116. For instance, although the subjectmatter described herein is not limited by any particular data structure,the data may be stored in computer registers, in a relational databaseas a table having many different fields and records, or XML documents.The data may also be formatted in any computing device-readable formatsuch as, but not limited to, binary values, ASCII or Unicode. Moreover,the data may comprise any information sufficient to identify therelevant information, such as numbers, descriptive text, proprietarycodes, pointers, references to data stored in other memories such as atother network locations, or information that is used by a function tocalculate the relevant data.

The computing device 110 may be at one node of a network 160 and capableof directly and indirectly communicating with other nodes of network160. Although only a few computing devices are depicted in FIG. 1, atypical system may include a large number of connected computingdevices, with each different computing device being at a different nodeof the network 160. The network 160 and intervening nodes describedherein may be interconnected using various protocols and systems, suchthat the network may be part of the Internet, World Wide Web, specificintranets, wide area networks, or local networks. The network mayutilize standard communications protocols, such as Ethernet, Wi-Fi andHTTP, protocols that are proprietary to one or more companies, andvarious combinations of the foregoing. As an example, computing device110 may be a web server that is capable of communicating with computingdevice 120 via the network 160. Computing device 120 may be a clientcomputing device, and server 110 may display information by usingnetwork 160 to transmit and present information to a user 135 of device120 via display 122. Although certain advantages are obtained wheninformation is transmitted or received as noted above, other aspects ofthe subject matter described herein are not limited to any particularmanner of transmission of information.

Computing device 120 may be configured similarly to the server 110, witha processor, memory and instructions as described above. Computingdevice 120 may be a personal computing device intended for use by a userand have all of the components normally used in connection with apersonal computing device such as a central processing unit (CPU),memory storing data and instructions, a display such as display 122(e.g., a monitor having a screen, a touch-screen, a projector, atelevision, or other device that is operable to display information),user input device 162 (e.g., a mouse, keyboard, touchscreen, microphone,etc.), and camera 163.

Computing device 120 may also be a mobile computing device capable ofwirelessly exchanging data with a server over a network such as theInternet. By way of example only, device 120 may be a mobile phone or adevice such as a wireless-enabled PDA, a tablet PC, a wearable computingdevice or a netbook that is capable of obtaining information via theInternet. The device may be configured to operate with an operatingsystem such as Google's Android operating system, Microsoft Windows orApple iOS. In that regard, some of the instructions executed during theoperations described herein may be provided by the operating systemwhereas other instructions may be provided by an application installedon the device. Computing devices in accordance with the systems andmethods described herein may include other devices capable of processinginstructions and transmitting data to and from humans and/or othercomputers including network computers lacking local storage capabilityand set top boxes for televisions.

Computing device 120 may include a component 130, such as circuits, todetermine the geographic location and orientation of the device. Forexample, client device 120 may include a GPS receiver 131 to determinethe device's latitude, longitude and altitude position. The componentmay also comprise software for determining the position of the devicebased on other signals received at the client device 120, such assignals received at a cell phone's antenna from one or more cell phonetowers if the client device is a cell phone. It may also include amagnetic compass 132, accelerometer 133 and gyroscope 134 to determinethe direction in which the device is oriented. By way of example only,the device may determine its pitch, yaw or roll (or changes thereto)relative to the direction of gravity or a plane perpendicular thereto.Component 130 may also include a laser range finder or similar devicefor determining the distance between the device and the surface of anobject.

Server 110 may store map-related information, at least a portion ofwhich may be transmitted to a client device. The map information is notlimited to any particular format. For instance, the map data may includebitmap images of geographic locations such as photographs captured by asatellite or aerial vehicles.

Server 110 may also store imagery such as, by way of example only, aflat photograph, a photo sphere, or a video of scenery. The imagery maybe captured and uploaded by end users for the purpose of making thephoto accessible for later access or to anyone searching for informationrelated to the feature. In addition to the image data captured by acamera, individual items of imagery may be associated with additionaldata, such as date of capture, time of day of capture, the geographicorientation (e.g., camera angle or direction) and location of capture(e.g., latitude, longitude and altitude).

Portions of the imagery may be associated with additional information,including a model of the geographic location of features that appearwithin the imagery. For instance, the model may identify the location ofsurfaces of objects captured in a panoramic image. The location of thesurfaces may be stored in memory in different ways, e.g., aconstellation of points whose location is defined as a solid angle anddistance from a fixed location (e.g., orientations and distances fromthe point from which the imagery was captured) or geographically-locatedpolygons whose vertices are expressed in latitude/longitude/altitudecoordinates. The system and method may further translate locations fromone reference system to another, such as generating a model ofgeographically-located polygons from a constellation of points whoselocations were captured directly with the use of a laser-range finder orgenerated from images by use of stereographic triangulation. Locationsmay be expressed in other ways as well, and depend on the nature of theapplication and required precision. By way of example only, geographiclocations may be identified by a street address, x-y coordinatesrelative to edges of a map (such as a pixel position relative to theedge of a street map), or other reference systems capable of identifyinggeographic locations (e.g., lot and block numbers on survey maps). Alocation may also be described by a range, e.g., a geographic locationmay be described by a range, or discrete series, oflatitude/longitude/altitude coordinates.

Example Methods

Operations in accordance with a variety of aspects of the invention willnow be described. It should be understood that the following operationsdo not have to be performed in the precise order described below.Rather, various steps can be handled in different order orsimultaneously.

A geographic object may be captured by imagery from multiple vantagepoints. By way of example, FIG. 2 shows a car 240 that was captured intwo separate images 215 and 225 from two different locations 210 and220, respectively. From vantage point 210, the camera angle 211 to aplane 202 generally defined by the front of the car is relativelyorthogonal, and the camera angle 212 to a plane 201 generally defined bythe side of the car is relatively acute. In contrast, from vantage point220, the angle of view 221 to front plane 202 is relatively acute, andthe angle of view 222 to side plane 201 is relatively orthogonal.(Reference numbers 201 and 202 are used interchangeably to refer to thefront and side surfaces of car 240 as well as the planes generallydefined by those surfaces.) In that regard, the images captured fromlocations 210 and 220 capture different surfaces of the car fromdifferent angles, with one surface being captured relatively straight onand the other being captured from a sharp angle.

A model of the location of the object surfaces captured in the imagerymay be generated and associated with the imagery. By way of example, thesource imagery may include a panoramic image. At the time a source imagewas captured, laser range finder 135 or another depth-determinationtechnique may have been used to associate each pixel of the panoramicimage with the distance from the capture location to the portion of asurface whose visual characteristics are represented by the pixel. Basedon such information and the location from which the image was captured(e.g., latitude/longitude/altitude information provided by GPS Received131), the system may generate a source model that includes a mesh ofpolygons (e.g., triangles) whose vertices correspond with the locationof points (e.g., latitude/longitude/altitude) along the surface of theobject.

The visual characteristics captured in the source image may be used totexture the model associated with that source image. For example, eachpixel of imagery 215 and 216 may also be associated with a camera angle,e.g., data defining a ray that extends from the camera to the portion ofthe surface associated with the pixel. The camera angle data may bebased on information provided by geographic component 130 at the timethe image was captured, such as the cardinal direction provided bycompass 132 and orientation data provided by gyroscope 134. Ray tracingmay be used to project the visual characteristics of each pixel of image215 onto the polygons of model 216 such that the visual characteristicsof the polygon at the point of intersection with the ray match thevisual characteristics of the associated pixel.

The visual information of the polygons may be stored as textures made upof texels. By way of example only, the texels of a texture may bearranged in a manner similar to the way that pixels of an image may bearranged, e.g., a grid or other collection of individual units thatdefine one or more visual characteristics. Portions of the followingdescription may refer only to a single visual characteristic of a pixelor texel, such as color, for ease of explanation. However, pixels andtexels may be associated with data defining many different visualcharacteristics including hue, saturation and brightness.

Even if two different source models include identical representations ofa surface's location, the visual characteristics of the surface in eachmodel may differ greatly. For instance, because imagery 215 captures theside 201 of car 240 at a sharp angle 212, the entire length of the sideof the car may horizontally appear in only a few pixels of image 215. Asa result, when the pixel information is projected onto polygonsrepresenting the side of the car, the information from a single pixelmay stretch across many texels in the horizontal direction. If model 216was displayed from vantage point 230, the side of the car may thusappear to have long horizontal streaks of colors as a result. However,if model 216 is displayed from vantage point 210 (the same location fromwhich the textures were projected), side 201 of car 240 might bedisplayed with few to no artifacts.

The system may combine visual information captured from multiple vantagepoints to display a geographic object from still another vantage point.For example, a user may request an image 235 of car 240 from vantagepoint 230. When determining the visual characteristics of the pixels ofthe image to be displayed to the user (“displayed pixels”), the systemmay associate each displayed pixel with a ray that extends from thevantage point and through an image plane containing the pixel. Thesystem may determine the point and texel at which each pixel'sassociated ray intersects a surface defined by the model. For instance,when determining the visual characteristics of a pixel in image 235, thesystem may determine the texel (T₁) at which the pixel's ray intersectsa polygon of model 216 and the texel (T₂) at which the pixel's rayintersects a polygon of model 226. The system may determine the color ofthe pixel by alpha-blending the color of T₁ and the color of T₂.

When the visual characteristics of the intersected texels are blendedtogether, greater weight may be applied to the texel derived from onesource image over a texel derived from another source image. The weightmay be based on the likelihood of the texel being an accuraterepresentation of the visual characteristics of the object to bedisplayed.

In at least one aspect, the quality value may depend on the resolutionof the displayed pixels relative to the resolution of the texels;optimum quality may be defined to occur when there is a single texel foreach displayed pixel. Conversely, low quality may be defined to occurwhen there are many texels associated with a single pixel or there aremany pixels associated with a single texel. By way of example, a firstpolygon may be generated wherein there is a single texel for each pixelprojected onto it. If the pixels were projected onto the first polygonfrom a relatively straight-on angle, the texture may include manytightly-packed texels. If this polygon was displayed straight on, theremay be roughly one texel for each displayed pixel and the texture wouldthus be considered to have a relatively high quality. However, if thepolygon was displayed from an acute angle, there may be many texels foreach displayed pixel and the texture would be considered to have arelatively low quality in spite of the texture's relatively highresolution.

By way of further example, a second polygon may be generated whereinthere is also a single texel for each pixel projected onto it. However,if the pixels were projected onto the second polygon from a relativelyacute angle, the texture may include long and thin texels. If thispolygon is then displayed straight on, many displayed pixels wouldderive their characteristics from only a single texel, in which case thetexture would be considered to have a relatively low quality. However,if the polygon is displayed from a relatively acute angle, there may beonly one texel for each displayed pixel, in which case the texture wouldbe considered to have a relatively high quality in spite of thetexture's relatively low resolution.

The quality value of a texel may be calculated based on the location ofthe user-defined and capture vantage points relative to the orientationof the surface to be displayed. By way of illustration, point 320 ofFIG. 3 may be the geographic location from which ageographically-located polygon of a model will be viewed. Point 330 isthe point at which the camera angle associated with a displayed pixelintersects the polygon. Line “s” extends from vantage point 320 tointersection point 330. Cone 350 extends outwardly from point 320 at anacute solid angle (“α”). The measure of a may be arbitrarily selected.The measure of a may also be chosen so that the cone provides a similarsolid angle as the texel as seen from the location from which thetexture was captured, e.g., the higher the texture resolution, thesmaller the cone. Ellipse 310 represents the intersection of cone 350and a plane (not shown). The plane reflects the orientation of thesurface at the intersection point, e.g., a plane defined by the verticesof the polygon that contains intersection point 330.

The extent to which the ellipse is stretched is related to theorientation of the surface and the vantage point. For example, if line“s” is perfectly orthogonal to the orientation of the plane (which wouldbe associated with viewing the surface straight on), ellipse 310 wouldbe a perfect circle. As the solid angle of line “s” becomes more acuterelative to the orientation of the plane, ellipse 310 will become morestretched and the ratio of the ellipse's major axis (“b”) relative tothe minor axis (“a”) will increase, with “n” as the surface normal. Axes“a” and “b” may be determined from the equation a=αs×n and b=(n·s/|s|)(a×n).

The quality value of a texel may be determined based on the differencesbetween the ellipse associated with a cone extending from the capturelocation to the intersection point (the “texel ellipse”) and the ellipseassociated with a cone extending from the location selected by the userto the intersection point (the “pixel ellipse”). In FIG. 4, texelellipse 425 is associated with capture location 420 and pixel ellipse435 is associated with user-requested vantage point 430. The qualityvalue of the texel at intersection point 450 may be calculated from theratio of the radius of the pixel ellipse relative to the radius of thetexel ellipse at a particular angle θ, e.g.,quality(θ)=radius_(t)(θ)/radius_(p)(θ). In one aspect, angle θ0 is theangle, or an estimate of the angle, that yields the minimum ratio. Forexample, the quality value may be calculated at different values of θ,and the texel's quality value may equal the lowest of the calculatedvalues, e.g., quality_(min)=quality (argmin_(θ){quality(θ)}). Theminimum may be determined by expressing both ellipses in matrix form andmultiplying the pixel ellipse with the inverse of the texel ellipse,which maps the pixel ellipse into a coordinate system where the texelellipse would be a unit circle. Within this coordinate system, the ratiois equal to the remapped pixel ellipse's radius and the minimum ratio isequal to the minor axis length of the remapped pixel ellipse.

The quality value may be estimated by projecting each of the texelellipse axes to each of the pixel ellipse axes, and selecting the pixelellipse axes that gives the largest ratio. By way of example, a shadermay approximate the minimum quality value by calculating values inaccordance with the equationquality_(min)˜1/max((a_(t)·a_(p))/(a_(p)·a_(p)),(b_(t)·a_(p))/(a_(p)·a_(p)), (a_(t)·b_(p))/(b_(p)·b_(p)),(b_(t)·b_(p))/(b_(p)·b_(p))), which samples four possible directions andselects the angle associated with the lowest quality value. Othermethods may also be used to calculate the quality.

Once the quality of a texel has been calculated, the quality may be usedto determine how similar the displayed pixel's color will be to thetexel's color. For instance, if three source images were used toidentify three texels T₁, T₂ and T₃, the blending weight for each texelmay be computed in a fragment shader for each input texel and output(w₁T₁+w₂T₂+w₃T₃)/(w₁+w₂+w₃), where w_(n) is equal to the quality valueof the texel. The weights may also be applied in other ways, such as byraising the quality values to a power. For instance, by raising theweights by a large exponent, the surface with the greatest weight maydominate the other weights, thus reducing ghosting by reducing theimpact of the texels with the lower weights.

The system may also use weight values to address occlusion, as shown byway of example in FIG. 5. Surfaces 502 and 503 were captured in image520 from vantage point A and surfaces 501 and 503 were captured in image530 from vantage point B. A model 521 and 531 may have been generatedfor each image 520 and 531, respectively. When the model is generated,the model may associate the texture with a specific side of the polygon.For example, surface 502 may be represented in model 521 by a singletriangle, where one side faces towards vantage point A and the otherside faces away from vantage point A. When the image data is projectedonto the model, the model may indicate whether the texture is on theside of the triangle facing towards the vantage point.

When generating the requested image, an occluded surface may be hiddenby determining whether its associated texture faces towards or away fromthe requested vantage point, and by determining whether the texture iscloser to the vantage point than other textures in the same model. Forinstance, when determining the color of a displayed pixel from vantagepoint 540, the system may identify all of the points at which thepixel's associated ray 550 intersects each surface, namely points511-513. The system may also determine, for each model, the intersectedtexel that is closest to the vantage point and whether the texture ofthe intersected texture is on the side of the polygon facing towards(“front facing”) or away (“back facing”) from the vantage point. Thus,the system may determine that T_(511B) is the closest texel in model 531and front facing (where “T_(pppm)” refers to the texel that is at thepoint of intersection ppp and stored in the model associated withvantage point m). The system may also determine that T_(512A) is theclosest texel in model 521 and back facing. Because T_(512A) is backfacing, the system may automatically set its weight to zero. As aresult, the color of the pixel associated with ray 550 may be determinedby (w_(511B)T_(511B)+w_(512A)T_(512A))/(w_(511B)+w_(512A)), wherew_(511B) is the quality value determined for T_(511B) and where w_(512A)is set to zero. As a result, the displayed pixel's color would be thesame as T_(511B).

Alternatively, rather than ignoring certain textures or setting theirweights to zero, their relative weights may be reduced. For example,rather than setting w_(512A) to zero and ignoring T_(513A) and T_(513B)altogether, the back-facing texels and other texels may also be used forblending but at reduced weights.

Artifacts may also be addressed with weight values, including artifactscaused by discontinuities in depth data. For example and as shown inFIG. 6, model 621 may be associated with the image captured fromlocation 620 and model 631 may be associated with the image capturedfrom location 630. The gap 603 between surface 601 and 602 may beaccurately represented in model 621. However, an inaccuracy arising outof the depth data retrieved at the time of capture, or some other errorthat occurred during generation of model 631, may result in model 631incorrectly indicating that a surface 635 extends across gap 603. Ifmodel 631 was viewed from vantage point 640, extraneous surface 635 mayhave the appearance of a rubber sheet stretched from one edge of the gapto the other. In some aspects, the system may check for and removeextraneous polygons by comparing model 621 with model 631.

In other aspects, the system may use extraneous surfaces when generatingan image for display. For instance, a user may request an image thatdisplays the surfaces from vantage point 640. When determining thecontribution of model 621 to the color of the pixel associated with ray641, the system may identify point A on surface 601 as the firstinstruction point. The system may also determine that the texel at pointA has a relatively high quality value because the texel is being viewedat an angle that is similar to the angle from which it was captured(location 620). When determining the contribution of model 631 to thecolor of the pixel, the system may identify point B on extraneoussurface 635 as the first intersection point. The system may determinethat the texel at point B has a relatively low quality value because thetexel is being viewed at an angle relatively orthogonal to the anglefrom which it was captured (location 630). As a result, when the texelsat points A and B are blended together, relatively little weight will beapplied to the texel at point B; the color of the pixel will be basedalmost entirely on the color of point A on surface 601.

From certain angles, the extraneous surfaces may have a significantcontribution to the visual characteristics of the image to be displayed.For example, a user may request an image from vantage point 650, whichis relatively close to the capture location 630 of model 631. Whendetermining the contribution of model 631 to the color of the pixelassociated with ray 641, the system may identify point C on extraneoussurface 635 as the first intersection point, and may further determinethat the texel at point C of model 631 has a relatively high qualityvalue. When determining the contribution of model 621 to the color ofthe same pixel, the system may identify no point of intersection ifsurface 602 is not represented by the model. Alternatively, if surface602 is represented by the model, the quality value of the texel at theintersection point in model 621 may be relatively low because the anglefrom the intersection point to vantage point 650 is relativelyorthogonal to the angle from the intersection point to capture location620. In either case and as a result, the color of the displayed pixelmay be substantially the same as the texel of the extraneous surface.This may be particularly advantageous if model 621 has no representationof surface 602, as displaying extraneous surface 635 may be preferablethan displaying nothing (e.g., a color indicating the absence of asurface).

Users may use the system and interact with the models. By way of exampleonly and with reference to FIG. 1, user 135 may use camera 163 andgeographic component 130 of client device 120 to capture image data anddepth data from multiple vantage points. User 135 may upload the imageand depth data to server 110. Processor 112 may generate a texturedmodel for each image based on the data provided by the user, and storethe data in memory 114. Server 110 may then receive a request for animage of the object from a specific vantage point. For example, user 135(or a different user using client device 121) may request a panoramicimage at a specific geographic location by selecting the geographiclocation with user input 162 and sending the request to server 110. Uponreceiving the request, server 110 may retrieve two or more models basedon the requested geographic location, such as by selecting all or alimited number of the models that have capture locations or surfacelocations within a threshold distance of the requested geographiclocation. The server may then send the models to client device 120 vianetwork 160. Upon receiving the models, the processor of client device120 may generate a panoramic image based on the models and using therequested geographic location as the vantage point. If the requestedgeographic location did not include an altitude, the altitude of thevantage point may be based on the altitudes of the capture locations ofthe models.

The panoramic image may be displayed on display 122. For example andwith reference to FIGS. 2 and 7, if models 216 and 226 of car 240 weresent to the client device, image 710 of car 240 may be generated basedon vantage point 230 and displayed to the user. The user may selectother vantage points that are different from the capture locations byproviding commands via the user interface of the client device, e.g.,pressing buttons to pan the image.

FIG. 8 is a flowchart in accordance with some of the aspects describedabove. At block 801, a model of the surface of an object is accessedthat includes: the orientation of the surface of an object relative to afirst vantage point and a second vantage point; a first set of visualcharacteristics representing the appearance of the surface from a firstvantage point; and a second set of visual characteristics representingthe appearance of the surface from a second vantage point. At block 802,a request is received for an image of the object from a requestedvantage point different from the first vantage point and the secondvantage point. At block 803, a first visual characteristic is identifiedfrom the first set of visual characteristics and a second visualcharacteristic is identified from the second set of visualcharacteristic. At block 804, a first weight value is determined for thefirst visual characteristic based on the orientation of the surfacerelative to the requested vantage point and the first vantage point. Atblock 805, a second weight value is determined for the second visualcharacteristic based on the orientation of the surface relative to therequested vantage point and the second vantage point. At block 806, avisual characteristic of the requested image is determined based on thefirst and second visual characteristics and the first and second weightvalues. At block 807, the requested image is provided for display to theuser.

As these and other variations and combinations of the features discussedabove can be utilized without departing from the invention as defined bythe claims, the foregoing description of the embodiments should be takenby way of illustration rather than by way of limitation of the inventionas defined by the claims. It will also be understood that the provisionof examples of the invention (as well as clauses phrased as “such as,”“e.g.”, “including” and the like) should not be interpreted as limitingthe invention to the specific examples; rather, the examples areintended to illustrate only some of many possible aspects.

1. A method of providing an image for display, the method comprising:accessing a first model of an object based on a first image of theobject captured from a first vantage point; accessing a second model ofthe object based on a second image of the object captured from a secondvantage point; receiving a request for a third image of the object froma requested vantage point different from the first vantage point and thesecond vantage point; generating the third image from the first modeland the requested vantage point, and from the second model and therequested vantage point; and providing the third image for display. 2.The method of claim 1, further comprising: determining a first visualcharacteristic from the first model and the requested vantage point; anddetermining a second visual characteristic from the second model and therequested vantage point, wherein generating the third image includesgenerating the third image to have a third visual characteristic that isbased on the first visual characteristic and the second visualcharacteristic.
 3. The method of claim 2, further comprising determiningan orientation associated with the requested vantage point, wherein thefirst visual characteristic of the first model is based on theorientation, and the second visual characteristic of the second model isbased on the orientation.
 4. The method of claim 2, wherein the thirdvisual characteristic of the third image is based on a first orientationassociated with the first model and a second orientation associated withthe second model.
 5. The method of claim 2, wherein the third visualcharacteristic of the third image is based on first location informationassociated with the first model and second location informationassociated with the second model.
 6. The method of claim 2, wherein: thefirst visual characteristic is based on a first ratio of a first texelcount to a first pixel count generated by projecting image data capturedfrom the first vantage point onto the first model; and the second visualcharacteristic is based on a second ratio of a second texel count to asecond pixel count generated by projecting image data captured from thesecond vantage point onto the second model.
 7. The method of claim 2,wherein: the first visual characteristic is a first texel generated byprojecting image data captured from the first vantage point onto thefirst model; and the second visual characteristic is a second texelgenerated by projecting image data captured from the second vantagepoint onto the second model.
 8. The method of claim 2, wherein the firstvisual characteristic and second visual characteristics are associatedwith colors, and wherein the third visual characteristic of the thirdimage is based on blending a first color associated with the firstvisual characteristic and blending a second color associated with thesecond visual characteristic.
 9. The method of claim 2, wherein theobject has a surface defining a plane, the method further comprisesselecting a point on the plane, and the first visual characteristic andthe second visual characteristic are determined based on a respectiveangle between the plane and a line extending from the first vantagepoint and the second vantage point to the selected point on the plane.10. A system comprising: one or more computing devices; and memorystoring instructions, the instructions being executable by the one ormore computing devices; wherein the instructions comprise: accessing afirst model of an object based on a first image of the object capturedfrom a first vantage point; accessing a second model of the object basedon a second image of the object captured from a second vantage point;receiving a request for a third image of the object from a requestedvantage point different from the first vantage point and the secondvantage point; generating the third image from the first model and therequested vantage point, and from the second model and the requestedvantage point; and providing the third image for display.
 11. The systemof claim 10, further comprising: determining a first visualcharacteristic from the first model and the requested vantage point; anddetermining a second visual characteristic from the second model and therequested vantage point, wherein generating the third image includesgenerating the third image to have a third visual characteristic that isbased on the first visual characteristic and the second visualcharacteristic.
 12. The system of claim 11, further comprisingdetermining an orientation associated with the requested vantage point,wherein the first visual characteristic of the first model is based onthe orientation, and the second visual characteristic of the secondmodel is based on the orientation.
 13. The system of claim 11, whereinthe third visual characteristic of the third image is based on a firstorientation associated with the first model and a second orientationassociated with the second model.
 14. The system of claim 11, whereinthe third visual characteristic of the third image is based on firstlocation information associated with the first model and second locationinformation associated with the second model.
 15. The system of claim11, wherein the first visual characteristic and second visualcharacteristics are associated with colors, and wherein the third visualcharacteristic of the third image is based on blending a first colorassociated with the first visual characteristic and blending a secondcolor associated with the second visual characteristic.
 16. The systemof claim 11, wherein: the first visual characteristic is a first texelgenerated by projecting image data captured from the first vantage pointonto the first model; and the second visual characteristic is a secondtexel generated by projecting image data captured from the secondvantage point onto the second model.
 17. A non-transitorycomputing-device readable storage medium on which computing-devicereadable instructions of a program are stored, the instructions, whenexecuted by one or more computing devices, causing the one or morecomputing devices to perform a method, the method comprising: accessinga first model of an object based on a first image of the object capturedfrom a first vantage point; accessing a second model of the object basedon a second image of the object captured from a second vantage point;receiving a request for a third image of the object from a requestedvantage point different from the first vantage point and the secondvantage point; generating the third image from the first model and therequested vantage point, and from the second model and the requestedvantage point; and providing the third image for display.
 18. The mediumof claim 17, further comprising: determining a first visualcharacteristic from the first model and the requested vantage point; anddetermining a second visual characteristic from the second model and therequested vantage point, wherein generating the third image includesgenerating the third image to have a third visual characteristic that isbased on the first visual characteristic and the second visualcharacteristic.
 19. The medium of claim 18, further comprisingdetermining an orientation associated with the requested vantage point,wherein the first visual characteristic of the first model is based onthe orientation, and the second visual characteristic of the secondmodel is based on the orientation.
 20. The medium of claim 18, wherein:the first visual characteristic is a first texel generated by projectingimage data captured from the first vantage point onto the first model;and the second visual characteristic is a second texel generated byprojecting image data captured from the second vantage point onto thesecond model.